Giving the Power to Bilingual Speakers Position paper for the APE Workshop

Post-editing has traditionally been used to improve the quality of MT output to make it usable. More recently, post-editing has also been shown to be useful for automatic evaluation of MT systems. As an alternative to current automatic evaluation metrics, such as BLEU and NIST, which are criticized for not having a high correlation with human judgments, minimal post-editing of MT output provides a good measure of how good a system translation is, and as an end result it also provides us with human reference translations that are relevant to MT systems. Such system friendly translations can then be used to automatically evaluate systems much more fairly, since systems are not penalized for having picked a different sense of the word or having picked a different, yet correct, syntactic construction (Snover et al., 2006). Mostly for practical reasons, post-editing has been a monolingual task done by linguists or editors without the source language information available. However, not having access to the original meaning can hinder the post-editing activity, and can even yield incorrect reference translations. In order to palliate this problem, for the GALE evaluation, multiple translators had to agree on a gold standard reference translation that conveys the exact same meaning as the source language sentence. This gold standard is then used by post-editors as if it were the original source language sentence when post-editing the raw MT output. Nevertheless, not only is the crafting of such a gold standard a difficult task, but in addition, there can be no guarantees that the real meaning of the source language sentence is fully captured by the gold standard translation. For language pairs with a large number of speakers, and certainly for the pairs researchers are working on for GALE (Olive, 2005), Chinese-English and Arabic-English, it would certainly be easier to get bilingual speakers to be post-editors, than having to generate a gold standard for each translation. In this context, I use the term bilingual speaker loosely. When I say bilingual speakers, I do not mean people that were born and raised bilingually and have native skills in both languages, but rather people that are native in one of the two languages and are fluent in the second language. A good example of what I mean by bilingual speakers here could be second generation Chinese and Arabic people that were born and live in the United States or another English-speaking country. …